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Abstract 

Computerized test construction by mathematical progrananing is 
possible given the availability an item bank calibrated under 
an IRT model . In this paper methods are proposed for the 
construction of weakly parallel tests, that is, tests with 
the same test information function. In the methods a 
mathematical programming model for constructing tests with a 
prespecified test information function and a heuristic for 
assigning items to tests such that their inforiration 
functions are equal play an important role* Numerical 
examples show that the tests are constructed very fast and 
that the heuristic gives good results. However, the heuristic 
is not applicable for every set of practical constraints, 
such as, constraints with respect to test administration 
time, test composition, or dependencies between items. 

Keywords: Item Banking, Test Construction, Weakly Parallel 
Tests, Mathematical Programming, Heuristics . 
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Methods and Models for the Construction 
of Weakly Parallel Tests 



In this paper an item bank is defined as a large^qllection 
of items calibrated under an item response model. Given, an 



item bank, tests can be constructed automatically by the 
application of mathematical programming models (Adema & van 
der Linden, 1989; Baker, Cohen & Barmish, 1988; Boekkooi- 
Timminga, 1989; van der Linden & Boekkooi-Tinuninga, 1989; 
Theunissen, 1985) . The goal of mathemat cal prograniming 
models is to optimize an objective function under a number of 
constraints. In mathematical programming models for test 
construction the goal is often to maximize the quality of the 
test under the constraint that the composition of the test is 
as required by the test constructor. A measure for the 
quality of a test is the test information function, because; 
the information function for a maximum likelihood estimator 
of ability is the reciprocal of the (asymptotic) sampling 
variance of the estimator (Lord, 1980) . Therefore, the test 
information function plays an important role in most 
mathematical programrdng models for test construction. Under 
the condition of local independence, the information function 
of a test can be computed by addition of the item information 
functions : 



N 

KG) - Z 11(6), 
i=l 



1. where N is the number of items in the test, 9 the ability 

I ' 

IT parameter, and 1^(9) the information function of item i. 

f ' 

f Boekkooi-Timminga (1987, 1990) nas proposed mathematical 

programming models for the construction of weakly parallel 

k 

I tests, where according to Samejima (1977) two weakly parallel 

test forms are "a pair of tests which measure the same 
V ability and whose test information functions are identical", 

i Two main parts of this paper can be distinguished. In 

J Part A it is assumed that a test information function is 

^ available and one or more tests should be constructed such 

^ that all tests have the same information function. In the 

i following section two methods are described for solving this 

! problem. In Part B of the paper two methods are given for 

selecting weakly parallel tests that are optimal with respect 
to the Maximin criterion. In this part first the criterion is 
formulated. Then, two methoas are proposed for solving the 
' problem. Next, some numerical examples are worked out to give 

an impression of the practicality of the proposed methods 
with respoct to CPU time and accuracy. 

; PART A: The Construction of Weakly Parallel Tests 

with a Prespecified Information Function 



In this part it is shown how one or more tests with a 
pr-jspecif ied test information function can be constructed. In 
the first section the case of one test is considered. In the 
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last two sections methods are given for the construction of ^; 
more than one test. These methods make use of the Minimax 
Model that is presented in the first section, 

A Mininax Mo del for the Construction of Weakly Parallel Tests 
In this section a mathematical programming raodel for the 
selection of items is given. It produces tests that 
approximate a prespecified test information function best ani 
in addition guarantees thau, for example, the number of items 
in the test and the composition of the test are as specified 
by the test constructor. Such a model can be used to select a 
test that is weakly parallel to a given test. 

The test information function is considered at a number 
of ability levels 6j^, k = 1,,,, K, The test constructor can 
choose the number and spacing of these ability levels. Let 
Ij^ (6)^) be the information value of item i at ability level 8)^ 
and T(6)^) the test information required at ability level 6)^, 
The decision variables Xj^ in the mathematical programming 
model, denoting if an item is or is not selected for the 
test, are define as follows: 



0 item i not in the test 

i = 1,- w I 

1 item i in the test. 
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The decision variables uj^ ienote the difference between T{Qy^) 
and the test information function value at 6)^ for the case 
that the latter is smaller than TCQ)^) : 

I 

Uk = max {0, TO)^) - 

If the test information function value at Qy^ is larger than 
T(6)^)/ the difference between the two is given by vj^: 

I 

v^ = max {0, I Ii(6k)Xi - TCe^)). 
k i=l 

The decision variable w is equal to the absolute value of the 
largest gap in information function value over all ability 
levels: 

w = max I U)^ + vj^ } 
k 

= max {max uj^; max V)^) . 
k k 

Given these definitions of th^i decision variables, the 
Minimax Model is formulated as: 

(1) minimize w, 

subject to 
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(2) uj^ + vj^ ^ w, k = 1,.., K, 



I 

(3) I Ii(Ok)Xi + ^k - ^k = T(ejc), k = 1,.., K, 

i=l 



I 

(4) I a^axi = b^, j ^ 1, J, 

i=l 



I 

(5) Z Xi = 



(6) Xi € {0,1), i « 1, I, 

(7) u^, vj^ ^ 0, k = 1, K, 

(8) w ^ 0. 

The objective of the model is to minimize the largest 
difference for all between the given test information 
function and the information function of the test to be 
constructed. The objective function (1) and the constraints 
(2) imply that the decision variable w is equal to the 
largest difference between T(6)^) and the test information 
function at ability level 6)^ for all 6)^. By constraints (3) 
uy - is equal to T(9j^) minus the test information function 
value at 9j^. The constraints presented in (4) are a general 
notation for practical constraints, that is, constraints with 
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respect to the composition of the test, the administration 
time of the test. etc. (see van der Linden & Boekkooi- 
Timminga, 19S3) . The number of items in the test is set equal 
to N by constraint (5) . 

The construction of, say, P weakly parallel tests can be done 
by applying model (l)-(8) P times. Every time the model has 
been applied, the selected items are deleted from the item 
bank so thc\t no item is contained in more than one test. 
Throughout this paper, this method of constructing weakly 
parallel tests will be called MI. As can be seen in the 
numerical examples below sometimes the psychometric quality 
of the tests will decrea^'S in the order in which they are 
constructed by method MI. In the next section a method is 
described that does not nave this drawback. 

Method MIDI 

This section is addressed to the problem of simultaneous 
construction of P weakly parallel tests with a prespecified 
test information function. The problem is solved here in two 
steps that will now be described. In the first step one large 
test is constructed that is P times the size of each of the 
tests to be constructed. In th** second I'^tep the items are 
assigned to the P tests such that the selected tests have the 
same composition and test information function. In the second 
step the heuristic DIFMIN is applied. This heuristic will be 
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described here for the case of practical constraints that 
imply a partition of the item bank. Consider for example, a 
mathematical item bank that can be partitioned into geometry 
and intermediate algebra items; one may determine that a test 
should contain 20 geometry and 20 intermediate algebra items. 
For other types of constraints DIFMIN should be adapted or is 
not applicable. 

DIFMIN is based on known heuristics for solving the 
"makespan" scheduling problem (e.g., Coffman, Lueker & 
Rinnooy Kan, 1988) . In this problem jobs have to be 
distributed among two or more parallel machines so as to 
minimize the time needed to process them. A good heuristic 
for the "makespan" scheduling problem according to 
probabilistic analysis (see, e.g., Bruno & Downey, 1986; 
Frenk & Rinnooy Kan, 1985; Loulou, 1984) is LPT (Largest 
Processing Time) . In LPT the jobs are sorted in order of 
decreasing processing time and, next, in decreasing order of 
processing time, each job is assigned to the machine for 
which the sum of the proces£?ing times of the jobs already 
assigned is smallest. In our problem the tests can be seen as 
machines and the items as jobs. Each item, however, does not 
have- one characteristic but K, namely th^ in format i ^n 
function values at 6)^, k = 1,.., K. This along with the 
practical constraints and the fact that the number of items 
should be equal for each test make the constructj-on of weakly 
parallel tests more difficult than the "makespan" scheduling 
pzoblem. As in LPT the heuristic starts with sorting the 
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items, that is, for each ability level Qy. the items are 
sorted in order of decreasing information function value. 
Then, the items are assigned to the tests on a one by one 
basis. In each iteracion of DIFMIN to each test an item is 
assigned under the restriction that these items be from the 
same subset of the partition of the item bank. The goal for 
edCh assignment is to close the largest gap in information 
function value for all tests and ability levels. 

In the description of the heuristic, Q denotes the Q-th 
iteration. After N iterations DIFMIN stops. 



Step 1 : Use Minimax (1) - (8) to construct a test that is P 
times the size of each of the P tests to be constructed with 
P*T(e)^) instead of Tfft» ) in (3), P*N instead of N in (5) and 
P*bj instead of bj in (4) . 

Step 2 ^Heuristic DIFMIN) 

Step 2a : For each ability level, 6)^, sort the items selected 

in Step 1 in order of decreasing information value. 

Step 2b : Initialization: The first item in the list of Qi is 

assigned to the first test (p=l) . Set Q equal to 1. 

Step 2c t Successively for tests p = 2 to P: Choose the item 

having the most information at B^, belonging to the same part 

of the item bank as the item selected in Step 2b, and not 

already assigned, and assign it to test p. 

Step 2d : If the number of items in each test is equal to N, 
that is, if Q = ^^ then STOP, else let Q := Q+1 and compute 
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max(k) = max Ip(6)^), k = 1,.., 

P 

where Ip(6)^) is the sum of the information function values at 
Qy^ Of the items already selected for test p» 
Compute the ability level Qmin' ^^^t Pj^in which has the 
largest gap in information function value with respect to all 
tests and ability levels: 

max(inin) - Ip (Qmin) = {max (k) - IpiQy^))* 

min p k 

The first item in the list of 6^^^^ assigned to a test 

is assigned to Pmin^ 

Step 2e : Compute Qmin Pmin such that 

max (min) - Ip (Qmin^ = max max {max (k) -Ip (6)^) } 
min P^'^Q^i ^ 

where ^q-i is the set of tests with Q-1 items. Select the 
first item in the list of 6^^^^^ that has not already been 
selected for a test and belongs to the same p*. rt of the item 
bank as the last item selected in Step 2d. Repeat this step 
until all tests contain Q items. 
Step 2f t Go to Step 2d. 
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Example: 

Suppose 6 items are selected in Step 1 and that these items 
should be assigned to two tests such that thei have 
(approximately) the same test information function at ability 



levels 6i 


and 62. 


The 


data are 


given by: 




Item 


1 


2 


3 


4 


5 


6 




0.8 


1.1 


2,0 


0.7 


0.5 


1.4 


1(62) 


0.6 


0.9 


0.4 


0.8 




1.1 


Subset 


G 


G 


G 


G 


lA 





where G stands for geometry ancJ lA for intermediate algebra. 
Th*- following lists are the result of Step 2a 

01 3 6 2 1 4 5 

02 5 6 2 4 1 3. 

;*jcording to Step 2b item 3 is assigned to test 1. Also, item 
2 is assigned to test 2 and not item 6 (Step 2c) , because 
items 3 and 6 belong to different subsets. The computations 
in Step 2d imply that the most informative item at 0^ should 
be selected for test 2. Item 3 was already selected; thus^ 
item 6 is selected for test 2. Because item 5 is the only 
item belonging to the same subset as item 6, this item is 
selected for test 1 (Step 2e) . After execution of Step 2ef 
Step 2d is executed again. The next item to be assigned 
should qive the most information at ability leve] 02 and is 
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selected for test 1. This implies that item 4 is assigned to 
test 1. Only item 1 remain and it is assigned to test 2. The 
results are as follows: 



Test Items 

1 3,4,5 

2 1,2,6 



T(9i) T(e2) 

3.2 2,4 

3.3 2,6 ar 



PART B: The Construction of Weakly Parallel Tests 
under the Maximin Criterion 



In this part two methods bacod on MI and MIDI are described 
for the construction of weakly parallel tests, where the 
te^ts are optimal with respect to the Maximin criterion. This 
criterion states th^it a test siiould give as much information 
as possible under the restriction that the test information 
function has the shape required by the test constructor and 
that the number of items is fixed. This criterion was 
proposed by van der Linden and Boekkooi-Timminga (1989) , As 
opposed to Part A, the test constructor now specifies a 
relative instead of an absolute target test information 
function. A brief introduction of the Maximin Model for the 
construction of one test will be given below. 



The test information function is considered at a number 
of ability levels Qy^, k = 1,,,, K, The number and spacing of 
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these levels should be specified by the test constructor. The 
relative shape of the target test information function should 
be specified by the test constructor by choosing rj^, 
k = Ir . .f K. Let y be a decision variable such that 
(r^yr.../ rj(y) is a series of lower bounds to the test 
information function. If N is the prescribed number of items 
in the test, then the Maximin Model is formulated as follows: 

(9) maximize y, 

subject to 



I 

(10) Z Ii(6k)Xi - rj^y ^ 0, k = 1,.., K, 

i=l 



I 

(11) I Xi = 
i=l 



(12) Xi e {0,1}, i = 1,.., I, 

(13) y ^ 0. 

The lower bounds are forced to be as high as possible by 
maximizing y in the objective function (9) . The constraints 
(10) are formulated such that (r^^y, rj^y) is a series of 

lower bounds to the test information function. The number of 
items in the test is controlled by constraint (11) . 
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k In the next two sections methods are proposed for the 

%' 

construction of weakly parallel tests, which are optimal with 

J,; 

B respect to the Maximin criterion. 

k 

I Method MAMI 

[ If the test information function of the weakly parallel tests 

is known the tests can be selected by applying method MI. 

I 

Method MAMI is based on the same idea. Two steps can be 

P 

i distinguished in the method. In the first step an estimate of 

I 

I the test information function is computed. In the second step 

method MI is applied. 



Step 1 ; A good approximation of the test information function 
is computed by constructing a large test with the Maximin 
Model that is P times the size of each of the P tests. By 
dividing the information function values of the large test at 
the specified ability levels by P, a target test information 
function is found for the Minimax Model. 



Step 2 : By applying the Minimax Model P times each time 
deleting itemt, selected at an earlier stage the P weakly 
parallel tests are constructed (Method MI) . 



Example : 

An item bank consists of 400 items. The first 200 items are 
of the multiple-choice type and the other 200 items are of 
the matching type. A test constructor wants to select items 
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for 3 weakly parallel tests each with 20 multiple-choice and 
20 matching items ♦ S{he) needs tests that measure well in the 
ability range [-2,2] and, therefore, the test information 
function is specified at the ability levels P « -2, 92 « 0, 
and 63 -2, and the corresponding values of ry^ are 
r^^ = r2 = r3 = 1. The Maximin Model is formulated as: 

maximize y, 

subject to 

400 

Z Ii(e)^)Xi - y ^ 0, k - 1, 2, 3, 

i=l 



200 

2 Xi = 60, 
i=l 



400 

I Xi = 60, 
i=20l 

Xi 6 {0, 1), i = 1, 400, 

y ^ 0, 

Let the information function values of the large test be 
L[Qy^)f k = 1, 2, 3; then the target test information function 
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is TCOj^) = L{ej^)/3* Thus, the following Minimax Model should 
be solved three times: 

minimize 

subject to 

U)^ + vj^ ^ k = 1, 2, 3, 



I Ii(ek)Xi + Uk - vj^ = TiQy,), k = 1, 2, 3, 

iel 



Z Xi = 20, 



I Xi = 20, 
ielM 



Xi e {0,1}, i € I, 

v,^ ^ 0, k - 1, 2, 3, 
w ^ 0. 

Every time a test is constructed the items already selectee^ 

are deleted from the item bank. Subset I is the set of items 

not deleted from the item bank. Subsets Ij^ and I^q are sets 

of items ^he matching and multiple-choice type not deleted 
from the item bank. 
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Met:hod MAPI 

The difference between method MIDI and MADI is that in the 
latter not the Minimax Model but th^ Maximin Model is used in 
Step 1 . 

5tep 1 ! As in laethod MAMI, method MADI starts with the 
construction of a test by the Maxim.ln Model that is P times 
the size of each of the weakly parallel tests. 

Step 2 ! Next, the items selected for the larc,e test are 
assigned to the P tests by the heurist*-: DIFMIN such that 
weakly parallel tests are created. 

Example: 

In this example the same problem as in ths last example is 
considered. First, a large test is constructed as in method 
MAMI. Heuristic DIFMIN then assigns the? selected items to the 
tests such that all three tests consist of 20 multiple-choice 
and 20 matching items and the test information functions are 
approximately equal. 

Numerical Examples 

In the examples below mathematical prograiiuning models 
had to be solved several times . The application of exact 
algorithms is too time consuming, therefore a heuristic has 
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been used. The applied heuristic solution strategy is in all 
cases so-called optimal rounding (van der Linden & Boekkooi- 
Timminga, 1989), because the practical constraints in the 
examples are such that always a feasible suboptimal solution 
will be found. Optimal rounding in this sense means, that 
first the relaxed model (0 ^ Xj^ ^ 1 instead of € {0,1}) is 
solved, then all variables x^ equal to 0 or 1 are fixed and, 
finally, the reduced model is solved to optimality by the 
branch-and-bound method (Land & Doig, 1960) . The optimal 
rounding method was implemented in ECL control programs for 
the software package MPSX/370 V2. ECL is a computer language 
based on PL\1. All methods considered in this paper were 
implemented in control programs such that they could be 
executed in one run. The execution times of the control 
programs are displayed in the tables below to show the 
practicability of the methods. The runs were executed on an 
IBM9370 computer. 

The item bank used in the examples is an existing 
Academic College Testing (ACT) item bank and was described by 
Ackerman (1989) . The bank consisted of 600 items; 520 items 
were from 13 previously administered ACT Assessment Program 
(AAP) tests and 80 were from the Collegiate Mathematics 
Placement Program (CMPP) . The items were calibrated using the 
3-parameter model (Birnbaum, 1968) . The bank was divided into 
six content areas: 

(1) Arithmetic and Algebraic Operations (AAO) ; 

(2) Arithmetic and Algebraic Reasoning (AAR) ; 
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(3) Geometry (G) ; 

(4) Intermediate Algebra (lA); 

(5) Number and Numeration Concepts (NNS) ; 

(6) Advanced Topics. 

From the bank items were selected to create weakly parallel 
tests with 40 items (4 AAO items, 14 AAR items, 8 G items, 
8 lA items, 4 NNS items, and 2 AT items) . 

Methods MI and MIDI 

The Minimax Model (1) - (8) can be applied to construct one 
^r more weakly parallex tests with a fixed target test 
information function. The methods MI and MIDI described above 
have been developed for this case. In tnis section the 
ability levels and the target information tunctj^n values 
(6^. TOj^)) are (-1.6, 2.0), (-.8, 5.4), {A 12.1), 

21.3), (1.6, 10.8), and (2.4, 3.1), respectively. 

The results for MI are displayed in Table 1. The number 
of tv?sts P is equal to 6. The test information function 
values are given for all the tests. 



Insert Table 1 here 



Figure 1 shows the 6 test information functions. 
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? Insert Figure 1 here 



i After the first 5 tests were constructed there was a 

t lack of "good" items • Therefore, the last test information 

I function having the lowest top, is not as close to the target 

I as the info at ion functions of the other tests. 

^ m Table 2 the test information functions of the tests 

constructed by MIDI are shown for P ranging from 2 to 6. 

i 

Insert Table 2 here 



An illustration of the test information functions for P « 6 
is given in Figure 2. 



Insert Figure 2 here 



The item bank contains a few items with extremely high 
difficulty. Therefore, it is possible again that the test 
information functions increase for ability levels outside the 
range considered. The tests were constructed simultaneously 
by MIDI and, therefore, there is no decrement in quality of 
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the tests in tt.e order of selection. Most of the CPU time was 
needed for the first step in the method. This stf.p was more 
time consuming for P = 6 than for P rang^.ig from 2 to 5. 
Thus, the total CPU time was larger for P « 6. 

MPthortfi MAMT and MAPI 

Two methods have been described for the construction of 
weakly parallel tests which are optimal with respect to the 
Maximin criterion: MAMI and MADI. In thi,, section the ability 
levels and the relative information function values (6)^, r)^) 
are (-1.6, 2.0), (-.8, 5.4), (.0, 12.1), (.8, 21.3), 
(1.6, 10.8), and (2.4, 3.1), respectively. 

The test information function values are shown in Table 
3 for the method MAMI for values of P, 2 through 6. 



Insert Table 3 here 



The test information functions for P = 6 are displayed in 
Figure 3. 



Insert ?igure 3 here 
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For all values of P, ranging from 2 to 6 the information 
function of the last test constructed was considerably worse 
than the information function of the other tests. The 
aberrant information function in Figure 3 is the information 
function of the last test. 

Tab?.e 4 shows the same results as Table 3, but now for 

MADI* 



Insert Table 4 here 



Figure 4 displays the information functions for P = 6. 



Insert Figure 4 here 



Comparing Figures 3 and 4, it can be seen that MADI does not 
suffer the drawback that later constructed tests tend to be 
worse than earlier constructed tests. This is because the P 
tests are constructed simultaneously. 
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Discussion 



In this paper the methods MI and MIDI were proposed for 
constructing tests with a prespecified test information 
function. Similar methods namely MAMI and MADI were given for 
the construction of we. Kly parallel test approximately equal 
with respect to the Maximin criterion. The four methods were 
applied on a real item bank. In the next two sections the 
practicality of the methods is discussed. 

Methods MI and MIDI 

The methods MI and MIDI can be used for the construction of 
weakly parallel tests with a fixed target test information 
function. Both methods can conscruct tests in a couple of 
minutes from a bank of 600 items. Thus, CPU time is no 
obstacle in using the methods. The method MI has the drawback 
that tests are selected sequentially, so that tests 
constructed later tend to be worse than earlier constructed 
tests. In Table 1 it is seen that this problem occurs ^or 
P = 6. The information function of the sixth test is not as 
good as the one of the other five tests in the closeness of 
the match of test information function and target information 
function For the MIDI method the test information functions 
are closer to the target test information functic than for 
the MI method, because the tests are constructed 
simultaneously. MIDI is also faster than MI. In MIDI the 
heuristic DIFMIN is used and this heuristic is described here 
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only for the case of practical constraints that imply a 
partition of the item bank. The drawback of MIDI is that the 
method is not applicable for other types of practical 
constraints • 

MP>tihods MA MT and MAPI 

The methods MAMI and MADI can be used to construct of weakly 
parallel tests with test information functions 
(approximately) optimal with respect to the Maximin 
criterion. If these methods are compared with respect to CPU 
time, it is clear that MADI is faster than MAMI, However, 
both methods can construct weakly parallel tests in a few 
minutes • The last test constructed by MAMI has always by fat 
the worst test information function, because the "good" items 
have already been selected. In this respect MAMI is not as 
good as MADI, Method MADI, however, suffers the same drawback 
as MIDI, 
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fe; Table 1 

I?' Information Functions for W ftakly Parallel Tests Constructed 
I Method MI 

I P Oi 62 63 64 05 06 

1 1.936 5.285 12.073 21.073 10.991 2.971 

^ 2 1.805 5.256 12.218 21.430 10.841 2.902 

3 2.037 5.788 12.456 21.568 10.389 3.072 

4 2.051 5.368 12.061 21.166 10.674 3.033 
I 5 1.898 5.371 12.278 21.446 10.717 2.853 

6 2.517 4.779 13.029 20.406 11.271 3.236 

; Hatfi. Total CPU-time: 1.97 minutes. 
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Table 2 

Information Functiop j f or Weakly Parallel Tests Constructed 
by Method MIDI 
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'f Table 3 

* Information Functions for W ^aklv Parallel Tests Constructed 

I by Method MAMI 



P 61 82 83 84 85 85 CPU 
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Table 4 

I nformation FunctionQ for Wpaklv Parallel Tests Constructed 



P 61 62 83 84 65 06 CPU 

(min) 
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2.47o 
2.510 
2.705 
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Figure Captions 



Figure 1 > Information functions 
method MI for P 6. 

Figure 2 . Information functions 
method MIDI for P = 6, 

Fi gure 3 > Information functions 
method MAMI for P = 6. 

Fi,gure 4 > Information functions 
method MADI for P = 6, 
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